Cluster families for cluster selection and cooperative replication

ABSTRACT

Cluster families for cluster selection and cooperative replication are created. The clusters are grouped into family members of a cluster family base on their relationships and roles. Members of the cluster family determine which family member is in the best position to obtain replicated information and become cumulatively consistent within their cluster family. Once the cluster family becomes cumulatively consistent, the data is shared within the cluster family so that all copies within the cluster family are consistent. Each family member in the family replicates 1/Nth volumes and N represents a total number of cluster family members in the family.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of U.S. patent application Ser. No.12/635,702, filed on Dec. 11, 2009.

FIELD OF THE INVENTION

This invention relates to data storage with respect to data storagesystems, and more particularly to clusters within storage systems.

DESCRIPTION OF THE RELATED ART

A storage system may include a plurality of tape drives that are used toaccess a plurality of magnetic tapes using a library manager. Themagnetic tapes may be disposed within cartridges. A controller maydirect an actuator to transfer a tape cartridge from a storage area totape drive in order to access data written on the magnetic tape and/orto write data to the magnetic tape.

Storage systems may be located at multiple sites including multiplegeographically distinct sites. The storage systems may communicate overone or more networks. Each storage system may include a plurality ofclusters. Each cluster may include a plurality of tape drives. Magnetictapes are mounted to the tape drives in order to read data from andwrite data to the magnetic tapes.

Each magnetic tape may be organized as one or more logical volumes,referred to herein as volumes. A volume may appear to a host as adistinct storage device. A volume may be logically “mounted” on avirtual tape drive. As used herein, a virtual tape drive is a logicalconstruct that appears to a host as a tape drive.

SUMMARY OF THE INVENTION

Methods, apparatus, and systems are provided to create cluster families,select clusters family members or families, and cooperatively replicateamong the family members and different families. For example, clustersare grouped based on their relationships into family members of acluster family. Members of the cluster family determine which familymember is in the best position to obtain outside data objects and becomecumulatively consistent with respect to outside data objects withintheir cluster family. Once the cluster family becomes cumulativelyconsistent, the data objects are shared within the cluster family sothat all clusters within the cluster family have a consistent copy ofeach outside data object. Each family member in the cluster familyreplicates 1/Nth volumes and N represents a total number of clusterfamily members in the cluster family.

BRIEF DESCRIPTION OF THE DRAWINGS

In order that the advantages of the invention will be readilyunderstood, a more particular description of the invention brieflydescribed above will be rendered by reference to specific embodimentsthat are illustrated in the appended drawings. Understanding that thesedrawings depict only typical embodiments of the invention and are nottherefore to be considered to be limiting of its scope, the inventionwill be described and explained with additional specificity and detailthrough the use of the accompanying drawings, in which:

FIG. 1 is a schematic block diagram illustrating an embodiment ofdistributed sites in accordance with the present invention;

FIGS. 2A and 2B are schematic block diagrams illustrating an embodimentof a storage system in accordance with the present invention;

FIG. 3 is a schematic block diagram illustrating an embodiment of acluster of the present invention;

FIG. 4 is a schematic block diagram illustrating an embodiment of acluster family apparatus of the present invention;

FIG. 5 is a schematic flow chart diagram illustrating an embodiment of acluster family selection and cooperative replication method of thepresent invention; and

FIGS. 6A and 6B are schematic flow chart diagrams illustrating anembodiment of a cluster family selection and cooperative replicationmethod of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

References throughout this specification to features, advantages, orsimilar language do not imply that all of the features and advantagesthat may be realized with the present invention should be or are in anysingle embodiment of the invention. Rather, language referring to thefeatures and advantages is understood to mean that a specific feature,advantage, or characteristic described in connection with an embodimentis included in at least an embodiment of the present invention. Thus,discussion of the features and advantages, and similar language,throughout this specification may, but do not necessarily, refer to thesame embodiment.

Furthermore, the described features, advantages, and characteristics ofthe invention may be combined in any suitable manner in one or moreembodiments. One skilled in the relevant art will recognize that theinvention may be practiced without one or more of the specific featuresor advantages of a particular embodiment. In other instances, additionalfeatures and advantages may be recognized in certain embodiments thatmay not be present in all embodiments of the invention.

This invention is described in embodiments in the following descriptionwith reference to the Figures, in which like numbers represent the sameor similar elements. While this invention is described in terms of thebest mode for achieving this invention's objectives, it will beappreciated by those skilled in the art that variations may beaccomplished in view of these teachings without deviating from thespirit or scope of the invention.

Many of the functional units described in this specification have beenlabeled as modules, in order to more particularly emphasize theirimplementation independence. For example, a module may be implemented asa hardware circuit comprising custom VLSI circuits or gate arrays,off-the-shelf semiconductors such as logic chips, transistors, or otherdiscrete components. A module may also be implemented in programmablehardware devices such as field programmable gate arrays (FPGAs),programmable array logic, programmable logic devices or the like.

Modules may also be implemented in software for execution by varioustypes of processors. An identified module of executable code may, forinstance, comprise one or more physical or logical blocks of computerinstructions, which may, for instance, be organized as an object,procedure, or function. Nevertheless, the executables of an identifiedmodule need not be physically located together, but may comprisedisparate instructions stored in different locations which, when joinedlogically together, comprise the module and achieve the stated purposefor the module.

Indeed, a module of executable code may be a single instruction, or manyinstructions, and may even be distributed over several different codesegments, among different programs, and across several memory devices.Similarly, operational data may be identified and illustrated hereinwithin the modules, and may be embodied in any suitable form andorganized within any suitable type of data structure. The operationaldata may be collected as a single data set, or may be distributed overdifferent locations including different storage devices.

Reference throughout this specification to “an embodiment,” “anembodiment,” or similar language means that a particular feature,structure, or characteristic described in connection with the embodimentis included in at least an embodiment of the present invention. Thus,appearances of the phrases “in an embodiment,” “in an embodiment,” andsimilar language throughout this specification may, but do notnecessarily, all refer to the same embodiment.

Furthermore, the described features, structures, or characteristics ofthe invention may be combined in any suitable manner in one or moreembodiments. In the following description, numerous specific details areprovided, such as examples of programming, software modules, userselections, network transactions, database queries, database structures,hardware modules, hardware circuits, hardware chips, etc., to provide athorough understanding of embodiments of the invention. One skilled inthe relevant art will recognize, however, that the invention may bepracticed without one or more of the specific details, or with othermethods, components, materials, and so forth. In other instances,well-known structures, materials, or operations are not shown ordescribed in detail to avoid obscuring aspects of the invention.

FIG. 1 is a schematic block diagram illustrating an embodiment ofdistributed sites 100 in accordance with the present invention. Thedistributed sites 100 include a plurality of sites 105. Each site 105communicates with the other sites 105 over a network 110. The network110 may be the Internet, local area network (LAN), wide area network(WAN), a dedicated network, a combination of networks, and the like.

Each site 105 may include one or more storage systems as will bedescribed hereafter. In addition, each site 105 may include bridges,routers, and the like that connect the storage systems to the network110.

FIGS. 2A and 2B are schematic block diagrams illustrating an embodimentof a storage system 200 in accordance with the present invention. One ormore storage systems 200 may be embodied in each site 105 of FIG. 1.

The storage systems 200 may store data in different physical media,including, but not limited to, storage cartridges, disk drives, solidstate disks (SSD), disks direct access storage devices (DASD), magnetictape drives, libraries, and disk drive arrays, such as RAID (redundantarray of independent disks), or JBOD (just a bunch of disks), An exampleof a storage cartridge is a magnetic tape cartridge, which includes arewritable magnetic tape wound on a hub of reel, and a cartridge memory.One example of a magnetic tape cartridge includes a cartridge based onLTO (Linear Tape Open) technology.

The storage systems 200 may store data in different forms, such aslogical or virtual data. Herein, data may be organized in any of variousforms, called “volumes” or “objects”, the terms chosen without referenceto any particular size or arrangement of data.

As illustrated in FIGS. 2A and 2B, the storage system 200 providesstorage for a plurality of host systems 210. For example, the storagesystem 200 includes a plurality of hosts 210, a plurality of clusters220, and a network 215. Although for simplicity, two (2) hosts 210 a,210 b, four (4) clusters 220 a, 220 b, 220 c, 220 d and one (1) network215 are shown in FIG. 2A, any number of hosts 210, clusters 220, andnetworks 215 may be employed. Accordingly, any number of clusters 220may be included in storage system 200.

As illustrated in FIG. 2A, the storage system 200 may employ four (4)clusters 220 a, 220 b, 220 c, 220 d connected by a network 215 with eachcluster 220 including a virtualization node (“VN”) 260 and a storagedevice 230 for emulating a tape drive or tape library to hosts 210 a,210 b. In an embodiment, clusters 220 a, 220 b, 220 c, 220 d are virtualtape server cluster.

Each cluster 220 includes a hierarchical storage node (“HSN”) 250 forlocally moving and/or transferring data between storage device 230 andlibrary 240. In an embodiment, storage system 200 includes a diskstorage 230 and a tape library 240. In an embodiment, the library 240 isan automated tape library (“ATL”). The HSN 250 may operate to remotelytransfer data between the local disk storage 230 and the remote diskstorage 230. The disk storage 230 may include one or more disk drivesarranged as a RAID, JBOD, SSD or any combination thereof, for example.

Each cluster 220 includes a library manager 370 as illustrated in FIG. 3with magnetic tapes as will be described hereafter. The hosts 210 mayinitiate and run tasks or jobs, such as tape jobs, in which data is readfrom and written to the magnetic tapes in the cluster families 280and/or family members 220. The hosts 210 may be mainframe computers,servers, or the like. The hosts 210 may have the ability to run or hostmultiple operating systems. For example, the hosts 210 may run or mayhost multiple operating systems such Linux, Java, Windows or the like.Each of the hosts 210 of the storage system 200 may operate as thesingle mainframe computer, one or more servers, or as number of virtualmachines. The hosts 210 may provide three levels of virtualizationthrough logical partitions (LPARs) via the PR/SM facility, throughvirtual machines via the z/VM operating system, and through operatingsystems, notably z/OS with key-protected address spaces andgoal-oriented workload scheduling.

The hosts 210 may communicate with the cluster 220 over the network 215to access a plurality of magnetic tape drives, disk drives, and otherstorage devices through the cluster family members 220 as will bedescribed hereafter. For example, a first host 210 a may communicateover the network 215 to access a storage device and a magnetic tapethrough a first cluster 220 a.

Each cluster 220 may include a hierarchical storage controller, such ashierarchical storage node 315, as illustrated in FIG. 3. The cluster 220may provide a single point management for data to be read and stored,aggregating storage pools in which storage can easily be allocated todifferent hosts 210, scaling the storage system 200 by adding storage orstorage control nodes, and a platform for implementing advancedfunctions such as fast-write cache, point-in-time copy, transparent datamigration, and remote copy.

The clusters 220 may follow an “in-band” approach. The in-band approachmay cause all input/output (I/O) requests and all management andconfiguration requests to be processed through a cluster family member220.

Each of the clusters 220 may be connected between themselves and withthe hosts 210 over the network 215 to access data written on themagnetic tape and/or to write data to the magnetic tape. The pluralityof clusters 220 may form a domain 205 of the storage system 200. Thedomain 205 may represent a multi-cluster or grid configuration. Thedomain 205 may include two or more clusters 220.

The network 215 of the storage system 200 may be storage area network(SAN), a token ring network, local area network (LAN), wide area network(WAN), the Internet, a dedicated network, a combination of networks, andthe like. The SAN may consist of a “fabric” through which the hosts 210may communicate with the clusters 220 over the network 215. The fabricmay include a Fibre Channel network, an Ethernet network, or the like.All elements may not share the same fabric for communication. The firsthost 210 a may communicate with the first cluster 220 a over one fabric.In addition, the first host 210 a may communicate with a third cluster220 c over another fabric.

Each storage system 200 may include a cluster family 280. The clusterfamily 280 may include a plurality of cluster family members 220 thatare arranged, configured, organized, and/or grouped into the clusterfamily 280 For example, as illustrated in FIG. 2B, storage system 200includes cluster family 280(1) and cluster family 280(2). Cluster family280(1) includes a plurality of cluster 220(a), 220(b) grouped intofamily members of cluster family 280(1). Cluster family 280(2) includesa plurality of cluster family members 220(b), 220(c) grouped into familymembers of cluster family 280(2). Cluster family 280(1) and clusterfamily 280(2) communicate with each via network, such as network 110,215. Each cluster family 280 may be given or assigned a name. Forexample, cluster family 280(1) may be named as City A and cluster family280(2) may be named as City B.

Although, for simplicity, FIG. 2B illustrates a storage system 200having two cluster families 280. Any number of storage systems 200,cluster families 280, and cluster family members 220 may be employed.

An example of a storage system 200 is the IBM® TS7700 Virtual TapeServer.

FIG. 3 is a schematic block diagram illustrating an embodiment of acluster 220 of the present invention. The cluster 220 may represent acluster family member 220 of cluster family 280 of FIGS. 2A and 2B, forexample. The description of cluster 220 refers to elements of FIGS. 1-2,like numbers referring to like elements. The cluster 220 may include avirtualization node 310, a hierarchical storage node 315, a volume cache365, and a library manager 370.

The storage device 230 may include one or more disk drives, for example,arranged as a redundant array of independent disks (RAID) or just abunch of disks (JBOD), or solid state disk (SSD), etc. The storagedevice 230 may include the volume cache 365. The volume cache 365 mayserve as a virtual volume cache and/or tape volume cache (TVC).

For example, storage device 230 includes a virtual volume cache 365. Thevirtual volume cache 365 may serve as a TVC, wherein the TVC includes arapidly accessible storage device such as a hard disk drive. In anembodiment, cluster 220 operates to cache data to the TVC 365.

The TVC 365 may cache data that is read from the logical volume and/orcache data that is to be written to the logical volume. A host 210 maymake repeated writes to a logical volume. The TVC 365 may store thewritten data on a hard disk drive 230 without writing the data to thelogical volume's magnetic tape. At a later time, the TVC 365 may writethe cached data to the magnetic tape within tape library 240.Accordingly, operations such as read operations and write operations fora virtual tape drive mounting a logical volume may be routed through theTVC 365.

A host 210 may initiate and run task and/or jobs on the cluster 220. Forexample, a first host 210 a access may result in an actuator of thelibrary manager 370 being controlled by a physical tape manager 335 totransfer a tape cartridge from a storage area to a tape drive in orderto access data written on the magnetic tape and/or to write data to themagnetic tape and/or TVC 365.

The virtualization node 310 may be an independent processor-based serverwith multiple connections to the network 215. The virtualization node310 may include either a battery backup unit (BBU) and/or may haveaccess to an uninterruptible power supply (UPS). The virtualization node310 may contain a watchdog timer. The watchdog timer may ensure that afailing virtualization node 310 that is not able and/or takes a longtime to recover may be restarted.

The virtualization node 310 may include one or more tape daemon 312. Thetape daemon 312 may emulate a tape drive of the cluster 220 to the host210 as a virtual tape drive. The tape daemon 312 may operate on a filethat is either on the TVC 365 and/or may operate on a file in a remoteTVC 365 of another cluster 220 through a remote file access 325.

The hierarchical storage node 315 may include a cluster manager 320, theremote file access 325, a data mover 330, the physical tape manager 335,a cache manager 340, a recall manager 345, a database 350, a managementinterface 355, and a media manager 360. The cluster manager 320 maycoordinate operations between the plurality of clusters 220 in amulti-cluster or grid topology.

The cluster manager 320 may use tokens to determine which cluster 220has a current copy of the data. The tokens may be stored in the database350. The cluster manager 320 may also coordinate copying data betweenthe clusters 220. The cluster manager 320 may include one or moreprocessors configured to execute computer readable programs as is wellknown to those of skill in the art.

The remote file access 325 may be a server, one or more processors, orthe like. The remote file access 325 may provide a link to the TVC 365for access by any remote cluster 220. The cluster manager 320 mayinclude a computer readable program.

The data mover 330 may control the actual data transfer operations forcopies performed between clusters 220 and also may transfer of databetween physical tape media and the TVC 365. The data mover 330 mayinclude a computer readable program.

The physical tape manager 335 may control the physical tapes in thecluster 220. The physical tape manager 335 may manage the physical tapesin multiple pools, reclamation, borrowing and returning of volumes fromand to a common scratch pool, and transfer tapes between pools. Thephysical tape manager 335 may include a computer readable program.

The cache manager 340 may control the copying of data from the TVC 365to the physical tapes and the subsequent removal of a redundant copy ofdata from the TVC 365. The cache manager 340 may also provide thecontrol signals to balance data flow between the different componentsand the TVC 365. The cache manager 340 may include a computer readableprogram.

The recall manager 345 may queue and control recall of data into the TVC365 from physical media for either a virtual tape drive or copiesrequested by the cluster manager 320. The recall manager 345 may includea computer readable program.

The database 350 may be a structured collection of records that may bestored on a hard disk drive. The records may include the locations ofdata on magnetic tape. The host 210 may write the data to the magnetictape of the cluster 220 and/or may access the data from the magnetictape using database addresses to provide the data to a user.

The management interface 355 may provide information about the cluster220 to the user. Also, the management interface 355 may allow the userto control and configure the cluster 220. The management interface 355may include a computer cathode ray tube (CRT), a liquid crystal display(LCD) screen, a keyboard, or the like, or exist as a web basedinterface.

The media manager 360 may manage the physical handling of the magnetictapes of the cluster 220. Also, the media manager 360 may manage errorrecovery of the magnetic tapes of the cluster 220. The media manager 360may diagnose errors and may determine if the errors are caused by thephysical tape drives or by the physical tape media. Further, the mediamanager 360 may take appropriate action for error recovery.

The library manager 370 may include plurality of physical tape drives, arobotic accessor, and a plurality of physical tape media. The roboticaccessor of the library manager 370 may transfer the magnetic tape to atape drive assigned to the TVC 365. A virtual tape drive may be alogical construct that appears to the host 210 as a physical tape drive.The data may be read from or written to the magnetic tape of the tapedrive through a read/write channel as is well known to those skilled inthe art.

Each tape drive of the plurality of clusters 220 may employ one or moremagnetic tapes to store the data. The magnetic tape may act as a storagemedia of the data in the storage system 200. The cluster 220 may employany number of tape drives and magnetic tapes. For example, the storagesystem 200 may employ two (2) tape drives and two hundred fifty six(256) virtual drives

The TVC 365 may contain data from tape volumes being operated on andstores additional volume data for rapid access. Operations such as readoperations and write operations for a virtual tape drive mounting avolume may be routed through the TVC 365. Thus selecting a cluster 220may select the cluster's TVC 365. All the magnetic tapes of the tapedrive may be organized as one or more logical volumes or volumes. Thevolumes in the TVC 365 may be managed using a first in first out (FIFO)and/or a least recently used (LRU) algorithm.

The TVC 365 may be a rapidly accessible storage device. For example, theTVC 365 may be a hard disk drive with a storage capacity of fivethousand four hundred gigabytes (5400 GB) or the like. In the storagesystem 200, the tape drive may cache data to the TVC 365 that is to beread from the logical volume and/or may cache data that is to be writtento the logical volume. For example, the host 210 may make repeatedwrites to a virtual tape drive. The TVC 365 may store the written dataon the hard disk drive without writing the data to the virtual magnetictape. At a later time, the cache manager 340 may write the cached datato the magnetic tape of the cluster 220.

The virtualization node 310 that accessed a volume may be referred to asa mount-point. Choosing a remote cluster TVC 365 that was used for arecent mount-point for a logical volume may improve access to thevolume. The high-availability, fast-write storage of the TVC 365 allowsthe hosts 210 to write data to the TVC 365 without having to wait forthe data to be written to a physical disk.

In an embodiment, each site 105 comprises a storage system 200. Eachstorage system 200 comprises two or more cluster family members 220grouped together to create a cluster family 280. For example, clusterfamily 280(1) comprises a group of cluster family members 220(a) and220(b) and cluster family 280(2) comprising a group of cluster familymembers 220(c) and 220(d). Cluster family 280(1) may be used forproduction purposes and cluster family 280(2) may be used for DR orarchival purposes, for example. Accordingly, cluster families 280 mayperform different roles with respect to other cluster families 280. Inaddition, cluster family members 220 of a cluster family 280 may performdifferent roles with respect to each other within the cluster family280. Accordingly, cluster family members 220 of a cluster family 280 mayperform different roles with respect to non-family members.

In an embodiment, cluster families 280 may be configured at globaldistances, metro distances, or combinations thereof. Similarly, clusterfamily members 220 may be configured at global distances, metrodistances, or combinations thereof. In addition, the cluster familymembers 220 may have different distant ratings from each other in acluster family 280. Similarly, cluster families 280 may have differentdistant ratings between each other. While distant ratings may be used asa factor to define roles and relationships between cluster families 280and cluster family members 220, this is but just a factor in bringingrelationship awareness between the cluster family members 220 andcluster families 280. Thus, arranging or grouping clusters 220 intocluster family members of a cluster family 280 is not limited todistances.

Additionally, because each storage system 200 includes a cluster family280 created by grouping two or more clusters 220 into family members,each storage system 200 or combination of storage systems 200 mayrepresent a multi-cluster configuration or grid.

Furthermore, the clusters 220 of storage system 200 may form distributedstore configuration. For example, a second cluster 220(b) may create asecondary instance of a volume. The secondary instance may besynchronized with the primary copy on a first cluster 220(a), whereinthe secondary copy is updated any time the primary copy is updated. Thesecondary instance may be stored in another cluster family 280 locatedat a remote site 105 in order to ensure availability of data in case theprimary instance becomes unavailable. Future mount-point accesses maychoose the secondary copy as the primary copy. Transparent datamigration may be used when adding, removing, and/or rebalancing data tomagnetic tape.

Although implementations of the present invention are discussed inreference to FIGS. 1-2, this is only for illustration purposes. Oneskilled in the art will appreciate that the present invention is notlimited to any specific grid configuration and may be implemented in anymulti-cluster or grid configuration. For example, one or more clusters220 from site 105(a) may be grouped with one or more clusters 220 from adifferent site 105, such as site 105(b), to create a first clusterfamily 280. Likewise, one or more clusters 220 from site 105(c) and site105(a) may be grouped in family members to create a second clusterfamily 280. Hence, any combination of clusters 220 may be grouped intofamily members to create a cluster family 280.

FIG. 4 is a schematic block diagram illustrating an embodiment of acluster family apparatus 400 of the present invention. The apparatus 400may be embodied in a host 210 and/or a cluster 220. In an embodiment,the apparatus 400 is embodied in the cluster manager 320. Thedescription of the apparatus 400 refers to elements of FIGS. 1-3, likenumbers referring to like elements. The apparatus 400 may include arelationship module 405, a creation module 410, a cooperativereplication module 415, a mount processing module 420, a communicationmodule 425, and a policy module 430 or any combination thereof.

The relationship module 405 comprises a computer readable programexecuting on a processor such as a processor of the cluster manager 320.In addition, the cluster relationship module 405 includes factorsdefining roles and relationship between cluster families 280 and familymembers 220. For example, factors relating to which family membersbelong to which families, the distance ratings between neighboringfamilies and/or family members, and which family members are used forproduction purposes and which ones are used for DR (disaster recover)and/or archiving purposes

The cluster family members 220 are in communication over a network suchas the network 110 and/or the network 215. Each cluster family member220 may comprise a library manager 370 with at least one tape driveconfigured to access volumes stored on magnetic tape and at least oneTVC 365.

The creation module 410 comprises a computer readable program executingon the processor such as the processor of the cluster manager 320. Thecreation module 410 selects and arranges clusters 220 into familymembers of a cluster family 280 by grouping clusters 220 together tooperate with a common set of guidelines, rules, and/or purpose.

The creation module 410 groups clusters 220 into a cluster family 280 toallow the family members 220 obey a common set of rules or guidelines.This allows groups of clusters, such as families 280(1), 280(2), forexample, to work together to accomplish a particular task moreefficiently or to allow different groups of clusters 220 an/or families280 to have different purposes within a grid.

The creation module 410 may be utilized to allow customizable behaviorof family members 220 within a family 280 through configurationproperties. For example, referring to FIG. 2B, a group of cluster familymembers 220(a), 220(b) may be allowed to act as production family 280(1)obeying a set of rules beneficial to production workloads. Another groupof cluster family members 220(c), 220(d) in the domain 205 may beallowed to act as an archival or disaster recovery family 280(2) withrules making family members 220(c), 220(d) operate more effectively inreplicating data from a production family 280(1).

In addition, the creation module 410 manages the relationships of familymembers 220 of a family 280 and the relationships between differentcluster families 280. For example, creation module 410 may managecluster family members 220 based on their relationships and roles. In anembodiment, the relationships module 405 may provide this information tocreation module 410. Based on the family members and neighboringfamilies' relationships and/or roles, the clusters family members 220will negotiate between each other to determine which family member 220is in the best position to obtain outside data from a plurality ofclusters outside of the family 280. The creation module 410 may also usethis information to favor members 220 of a family 280 as TVC clusters orallow access restrictions or other special case behavior on a family 280as opposed to just a cluster or grid-wide.

Creation module 410 may utilize the management interface 355 to displaya page where a user (e.g., customer) may create a cluster family with acharacter name, such as an eight character name. The user may then addone or more clusters to a family using the creation module 410. Creationmodule 410 may store this information within a cluster persistent vitalproduct data so that all clusters in a multi-cluster or gridconfiguration are aware of their cluster's role and the family itresides in. The creation module 410 may determine that a cluster beingselect for a family is already selected for another family. To avoidhaving any one cluster existing in two families at the same time, thecreation module 410 may notify the user that the cluster being selectalready exist in another family member. In addition, the creation module410 may employ a set of rules to prevent the selection of one clusterinto two families at the same time.

The policy module 430 comprises a computer readable program executing onthe processor such as the processor of the cluster manager 320. In anembodiment, the policy module 430 may include certain policies relatingto which cluster family members 220 should be used for production andwhich family members should be used for DR/archival purposes. Thesepolicies may include sets of rules governing the replication of data. Auser may enter the policies for managing multiple cluster families 280and family members 220 via management interface 355.

Referring to FIGS. 2A and 2B, cluster family creation module 410 may beused to create a cluster family 280(1) named “City A” and to createanother cluster family 280(2) named “City B”. Cluster family 280(1) mayinclude a group of cluster family members 220(a), 220(b) and clusterfamily 280(2) may include a group of cluster family members 220(c),220(d). In addition, the creation module 410 may be used to add orremove family members 220 to or from a cluster family 280 and to regroupcluster family members into different cluster families 280.

Because the creation module 410 sets up and arranges clusters intofamily groups based on their relationships and/or roles to each otherduring the creation of a family, all clusters in a grid or multi-clusterconfiguration are aware of each others roles and the families theyreside in. Hence, the creation module 410 may alert or notify a user viamanagement interface 355 that the cluster 220(d) being added to thefamily 280(1), for example, is currently a family member of anotherfamily 280(2). The user may then deselect 220(d) from family 280(2) andadd or reselect 220(d) to family 280(1). Accordingly, the creationmodule 410 allows all clusters 220 in a domain 205 (e.g., a grid) to beaware of their own role and relationship to their family they reside in,to other family members, and to non-family members residing in otherfamilies.

In an embodiment, the creation module 410 may assign a name to a clusterfamily. For example, during configuration a user may assign a name to acluster family using the management interface 355.

The cooperative replication module 415 comprises a computer readableprogram executing on the processor such as the processor of the clustermanager 320. In addition, the cooperative replication module 415enhances existing copy management to enable groups of clusters 220belonging to a cluster family 280 to work together to be more efficientin achieving consistency for the family 280 as well as among individualclusters 220 within the family 280 (e.g., family members 220).

The cooperative replication module 415 allows two or more cluster familymembers 220 within a family 280, such as a DR or archival family, toshare inbound replication workload. Accordingly, a family 280 ofDR/Archival cluster family members 220 utilizing cooperative replicationmodule 415 benefits from improved TVC selection when choosing a sourcecluster for replication.

The cooperative replication module 415 allows a cluster family member toshare a copy workload among other cluster family members belonging tothe same family. For example, in an embodiment, a domain 205 includes Yclusters 220, where Y represents the number of clusters 220 included inthe domain 205. The clusters are grouped into cluster families 280having two or more cluster family members 220. Hence, the domain 205 ismade up of Y clusters 220 in which some of the clusters 220 are groupedinto N cluster family members of a cluster family 280.

For example, referring to FIG. 2B, there are four clusters 220(a),220(b), 220(c), and 220(d) in domain 205 so Y represents four clusters(Y=4). Two clusters 220(a) and 220(b) are grouped into N cluster familymembers of a first cluster family 280(1) and two clusters 220(c) and220(d) are grouped into N cluster family members of a second clusterfamily 280(2). In this grid configuration, domain 205 is made up of Y(4)clusters, in which a subset of N clusters are grouped into familymembers of cluster families 280. Accordingly, N=number of cluster familymembers in a family.

The cooperative replication module 415 cooperatively replicates a familygroup of clusters by serializing the replication of any one volume whenbringing it into the family for the first time. For example, thecooperative replication module 415 directs each cluster member 220(c)and 220(d) in the family 280(2) to replicate 1/Nth of outside volumeswhere N is the number of clusters in the family requiring a copy. Onceall outside volumes are replicated into the family 280(2) and the family280(2) is cumulatively consistent, the inconsistent clusters within thesame family 280(2) then share among each the outside data.

As an example, it is possible from a microcode level that each cluster220 works independently of each other because the clusters 220 areunaware of there relationships and roles to each other. For example, ifwe assume that cluster 220(a) includes 20 volumes that need to bereplicated to clusters 220(c) and 220(d). Because clusters 220(c) and220(d) are working independently of each other, each cluster 220(c),220(d) may pull 20 copies of the original data across network 215.

Now referring to FIGS. 2A and 2B, in an embodiment, for example, thereare four clusters in which two clusters 220(a), 220(b) are grouped intofamily 280(1) and two clusters 220(c), 220(d) are grouped into family280(2) via creation module 410. All family members 220 are aware of eachother and the families 280 they belong to and are aware of all thevolumes needing to be replicated into adjacent clusters within families.

For example, cluster family member 220(c) and 220(d) are aware of eachother and that there are 20 volumes from a non-family member 220(a)within a different cluster family 280(1) that needs to be replicatedinto their family 280(2). Utilizing the cooperative replication module415, family member 220(c) pulls 10 unique volumes and family member220(d) pulls the other 10 unique volumes. That is, each cluster familymember 220(c), 220(d) pulls 1/Nth of the volumes, where N=number ofcluster family members in a family. Because in this example there aretwo cluster family members 220(c), 220(d) belonging to family cluster280(2), each family member pulls ½ of the volumes (e.g., each pulls 10unique volumes) to get a total of 20 volumes. The cluster family members220(c), 220(d) then share the 10 unique volumes with each other.

By cooperatively replicating via the cooperative replication module 415,the cluster family 280(2) or DR location may become cumulativelyconsistent N time faster because any one volume was only pulled acrossthe distant link 110/215 once versus N times. The cluster family members220 may then become consistent among each other for availability muchfaster due to their relative distance between them. Accordingly, theoverall time to become both DR consistent and highly available (HA)consistent may be greatly enhanced over each cluster 220 independentlysourcing from the same remote production clusters 220.

Accordingly, it is possible to optimize copy throughput and improve theoverall time to reach volume consistency within the cluster family 280.For example, in a limited bandwidth system or a grid with multiplearchive sites, the cooperative replication module 415 allows each familymember 220 in a family 280 to participate in the replication process forall inbound copies without duplicating any effort. Once a group ofclusters (family members) 220 within a family 280 reach an aggregateconsistent state, the consistent copies within individual clusters 220in the family 280 are shared among peer clusters within the same family.

In addition, the cooperative replication module 415 handles persistentreplication source awareness by deferring replication. For example, acluster member 220 that has a consistent source may be instructed tomaintain that source volume in cache in order to make it readilyavailable to other family members 220 for peer replication. The clusterwith the consistent source inherits the role of the originalmount-source cluster or the cluster containing the host created/modifiedoriginal copy. Once one cluster in a family replicates one of its 1/Nthvolumes, the cooperative replication module 415 first informs theoriginal mount-source cluster that all other clusters in its family,including itself, are accounted for and that the production cluster canrelieve itself of the role on behalf of the clusters in the targetedfamily. This frees the production cluster to stage the volume out toback end tape (assuming there is no other families or productionclusters needing copies), thus providing more cache availability.Second, the DR family cluster which initiated the replication for thevolume inherits the role and remembers which clusters within its familystill need a copy. Through this inheritance, the volume may be favoredin cache until all its peer family clusters have completed a copy.

In an embodiment, the cooperative replication module 415 may employcascading copy required flags. For example, cooperative replicationmodule 415 moves the ownership of the copy required flag from onecluster family to another as a cluster family becomes consistent. Bycascading the copy required flags, the cooperative replication module415 may allow the benefits of the flags to be shifted from one family toanother thus relieving the original TVC of its involvement. Byinheriting the copy required flags from the TVC, for example, once afamily member attained a copy it may allow the TVC cluster to migratethe volume and make room in cache for other new workload.

An example may be a domain consisting of a production or default familyin conjunction with a DR/archival family. The TVC cluster may be amember of the production or default family and may begin managing thecopy required flags. Once a member of the DR/archival family obtains acopy from the TVC cluster the DR/archival family may notify the TVCcluster to clear all copy required flags pertaining to members of theDR/archival family. In conjunction with this, the DR/archival family mayinherit the responsibility of managing those copy required flags for itsfamily members.

For example, domain 205 may include a first family cluster 280(1)comprising cluster family members 220(a), 220(b), 220(c), a secondfamily cluster 280(2) comprising cluster family members 220(d), 220(e),220(f), and a third family cluster 280(3) comprising cluster familymembers 220(g), 220(h), 220(i). Each family 280 includes three clusterfamily members 220 and each family member represents a bit. There is atotal of 9 bits in the bit-set because there are three families witheach family having three family members (3 bits). Family cluster 280(1)includes the original data object needing to be replicated into familyclusters 280(2), 280(3), for example.

It is possible that cluster 220(a) may hold the volume in cache untilall nine clusters 220 have pulled a copy across the network 110 or 215.For example, cluster 220(a) may include a 9 bit-set and when eachcluster 220 pulls a copy, cluster 220(a) may clear a bit in its mask.Because cluster 220(a) is holding a copy in its cache for all nineclusters 220, cluster 220(a) may not be able to make room for additionalworkload.

By allowing each cluster family 280 to inherit the responsibility ofmanaging those copy required flags for its family members, cluster220(a) may clear the remaining 6 bits for those cluster family 280(2),280(3), and only keep the copy in its cache for its own two familymembers 220(b), 220(c) residing in family 280(1). Once its own familymembers 220(b) and 220(c) have a copy, cluster 220(a) may then clear itsmask to make room for more workload.

In this example, cluster 220(d) of family 280(2) pulls a copy acrossnetwork 215 and informs cluster 220(a) of family cluster 280(1) that itno longer needs to hold the copy in cache for family 280(2) because220(d) will keep a copy in its cache until its family members 220(e),220(f) receive a copy. This relieves cluster 220(a) from holding thecopy in cache for all the family members of cluster family 280(2).Similarly, a family member 220(g) belonging to family 280(3) instructscluster 220(a) that it will keep a copy in its cache for its familymembers 220(h), 220(i). Thus, cluster 220(a) is relieved of holding acopy in its cache for all the family members belonging to 280(3).

In addition, the cooperative replication module 415 may increaseperformances in low bandwidth environments by utilizing more linkswithin the domain to perform copies instead of primarily relying oncopies from the TVC and the overall time for clusters within a family tobecome consistent improves. For example, families 280 employing thecooperative replication module 415 collaborate in order to achieveconsistency across the family. Upon reaching family-wide consistency,the family members 220 then work together to share data amongst familymembers to bring each individual member up to the family's consistencylevel.

The mount processing module 420 comprises a computer readable programexecuting on the processor such as the processor of the cluster manager320. The mount processing module 420 favors and selects cluster familymember in its own family over clusters outside its family when a mountoccurs to a logical volume with a cluster. For example, a mount to aproduction cluster may favor another production cluster in the samefamily 280(1) over a remote cluster being used primarily for DR orelectronic vaulting. The mount processing module 420 may be employed tofavor availability over disaster recoverability when production dataneeds to remain local and replicate quickly for high availability andthus, select family members within the production family over the DRfamily.

The mount processing module 420 may improve control and performance byfavoring cluster family members when remote mounts are required.Families and/or family members may be configured (e.g., using thecreation module 410) to prefer certain clusters over other clusters whenchoosing a remote TVC. This may be beneficial in distinguishing a set ofproduction clusters from non-production clusters. Preferring the familymembers within the same production family may keep TVC selection withinthe production clusters rather than potentially choosing distant remoteclusters intended for DR or archival purposes.

In addition, because the cluster family member 220 within a clusterfamily 280 is the target of a mount and is favored within the samecluster family 280, the TVC selection processing may be improved.

In an embodiment, a storage system 200 may include a plurality ofclusters in which a subset of two or more clusters 220 are grouped intoa first cluster family 280 and a subset of two or more clusters 220 aregrouped into a second cluster family 280. The grouping of the familygroup may be based on the family members' roles, relationship, and/ordistances to each other and/or to other non-family member clusters. Eachcluster family member of the cluster family 280 is aware of theirrelationship to each other. This relationship awareness among familymembers allows the group to work together effective to cumulativelyreplicate data into the group and then replicate amongst each other.

Site 105 may include a cluster family 280 or a combination of clusterfamilies 280. For example, site 105(a) may include a first clusterfamily 280 and a second cluster family 280. The first cluster family 280may include production clusters 220(a), 220(b) and the second clusterfamily 280 may include DR clusters 220(c), 220(d). In addition, clusters220 may be selected from a combination of sites 105 to create a clusterfamily 280. For example, a cluster family 280 may be created fromselecting clusters 220 at multiple sites 105, such as 105(a) and 105(b),wherein clusters 220(a), 220(b) at site 105(a) are used for productionpurposes and clusters 220(c), 220(d) at site 105(b) are used for DRand/or achieving purposes.

In an embodiment, clusters 220(c), 220(d) are used for archiving data.In an embodiment, clusters 220(c), 220(d) are used for DR. In anotherembodiment, one cluster, such as cluster 220(c), is used for DR and theother cluster, such as cluster 22(d), is used for archiving.

The schematic flow chart diagrams that follow are generally set forth aslogical flow chart diagrams. As such, the depicted order and labeledsteps are indicative of an embodiment of the presented method. Othersteps and methods may be conceived that are equivalent in function,logic, or effect to one or more steps, or portions thereof, of theillustrated method. Additionally, the format and the symbols employedare provided to explain the logical steps of the method and areunderstood not to limit the scope of the method. Although various arrowtypes and line types may be employed in the flow chart diagrams, theyare understood not to limit the scope of the corresponding method.Indeed, some arrows or other connectors may be used to indicate only thelogical flow of the method. For instance, an arrow may indicate awaiting or monitoring period of unspecified duration between enumeratedsteps of the depicted method. Additionally, the order in which aparticular method occurs may or may not strictly adhere to the order ofthe corresponding steps shown.

FIG. 5 is a schematic flow chart diagram illustrating an embodiment of acluster family selection and cooperative replication method of thepresent invention. The method 500 substantially includes the steps tocarry out the functions presented above with respect to the operation ofthe described apparatus and system of FIGS. 1-4. In an embodiment, themethod is implemented with a computer program product comprising acomputer readable medium having a computer readable program. Thecomputer readable program may be integrated into a computing system,such as the cluster manager 320 and/or hosts 210, wherein the program incombination with the computing system is capable of performing themethod 500.

The method 500 starts and in step 510, a group of clusters are arrangedinto family members of a cluster family. For example, clusters aregrouped base on their relationships to each other and other clusters ina domain. Cluster families may be created based on a variety of factorsand/or functions including roles (e.g., production source, DR,archiving, etc), scope, distances (e.g., distance ratings betweenfamilies), and the like. In addition, a user may assign a character nameto create a cluster family. For example, as illustrated in FIG. 2B, acluster family may be created using a character name “City A” andanother family may be created using a character name “City B”.

In an embodiment, the creation module 410 is used to create a clusterfamily, wherein a user may create a cluster family during configurationusing a management interface 355 to create a cluster family name, addone or more clusters to a family, assign a role and/or a distance ratingbetween neighboring families, and educate the clusters usingconfiguration properties. These persistent settings may be used by thecreation module 410, for example, to bring relationship awareness to oneor more clusters or family members as well as relative propertiesbetween families, such as distance.

In an embodiment, the relationship module 405 maintains these persistentsettings for the cluster families and family members.

Additionally, an autonomic functionality may be employed to detect theroles and relationships between clusters. The autonomic functionally matbe executed in creation module 410, for example.

In step 515, family members negotiate between each other to determinewhich family member of the family which family member is in the bestposition to obtain outside data objects. For example, as illustrated inFIG. 2B, cluster family 280(1) includes two or more family members220(a), 220(b), which may be configured at metro distances and used forproduction purposes. Cluster family 280(2) includes two or more clusterfamily members 220(c), 220(d), which may be configured at globaldistances with respect to family 280(1) and used for DR purposes.Cluster family 280(1) may communicate with cluster family 280(2) vianetwork 110, 215 that data objects are ready to be copied. Becauseclusters members of each family as well as the families themselves areaware of each others roles and relationships to each other, familymembers 220(c), 220(d) may negotiate between each other to determinewhich family member of the family 280(2) is in the best position toobtain a copy of the outside data objects.

In an embodiment, for example, cluster family members 220 belonging to acluster family 280 work in FIFO order using a common copy work queue.Before working on a copy, each cluster family member 220 first makessure no other cluster family member in the cluster family 280 is alreadycopying or has already copied. If not, one or more cluster familymembers 220 may perform the copy. If copying is occurring by anotherfamily member or has already occurred by another family member, one ormore cluster family members may move the copy into a deferred queue.Sometime later after all active production content is copied into thecluster family 280, the family members start working on the deferredqueue which is content they should be sharing among each other. If thepeer family member that originally got the copy isn't present, it maystill get a copy from an outside cluster or another family member.

In step 520, one or more cluster family member obtains and replicatesthe information or source volume. For example, one or more clusterfamily members 220 belonging to a cluster family 280 is selected to pullthe data or source volume across the remote network 110,215copy/replicate the date or source volume, and bring it into the clusterfamily 280. For example, the family member 220(c) of family 280(2) pullsthe outside data objects into the family 280(2) over network 110, 215.Family member 220(c) now has a consistent source and may be asked tomaintain that source volume in cache (e.g., TVC 365) to make it readilyavailable for peer replication.

In step 525, the source volume is cooperatively replicated among familymembers of the family. For example, a family group of clusters willcooperate by serializing the replication of any one volume when bringingit into the family for the first time. The clusters in a family may eachplay a role in replicating 1/Nth of the volumes where N is the number ofclusters in the family requiring a copy.

By cooperatively replicating, the cluster family or DR location maybecome cumulatively consistent N times faster since any one volume wasonly pulled across the distant link once versus multiple times. Theclusters can then become consistent among each other for availabilitymuch faster due to their relative distance between them. The overalltime to become both DR consistent and HA (High availability) consistentmay be greatly enhanced over each cluster independently sourcing fromthe same remote production cluster.

In step 530, the cluster family achieves cumulative consistency. Thatis, all volumes outside of the cluster family that need to be replicatedinto the cluster family are completed. The cluster family as a whole isconsistent with respect to all outside data objects. Now, the clusterfamily members may share among each other so that each individual familymember within the cluster family has its own copy.

In step 535, after all volumes are replicated into a family and thefamily is cumulatively consistent, the inconsistent clusters within thesame family then share volumes (i.e., data objects) among each other.

Accordingly, implementing method 500 of the present invention performscooperatively replication, in which the cluster family or DR locationcan become cumulatively consistent N times faster since any one volumewas only pulled across the distant link once versus N times. Theclusters may then become consistent among each other for availabilitymuch faster due to their relative distance between them. The overalltime to become both DR consistent and HA consistent may then be greatlyenhanced over each cluster independently sourcing from the same remoteproduction cluster.

In addition, method 500 uses families for a more effective method ofreplicating to X clusters when only N copies are required by thecustomer and those N copies must be distant from each other. This allowsa customer to spread copies across distances/families without being overexplicit in which clusters receive a copy. For example, a user may nothave a concern about which clusters contain the copy so long as N copiesexist (where N is less than X); and, the customer demands that the Ncopies all exist in independent families. Therefore, all clusters in adomain may cooperate to make sure at least one member from each familyreplicates a volume and the remaining clusters may then surrender itsreplication requirements. It is possible then to end up with N copies inN families without having too many of the N copies in any one region.

The steps of method 500 may be employed in a mount processing in anycombination thereof. For example, with cluster families configured instep 510, using steps 515-535, method 500 may favor clusters in its ownfamily over clusters outside its family. For example, a mount to aproduction cluster may favor another production cluster (in the samefamily) over a remote cluster used primarily for disaster recovery(electronic vaulting). Since a user may tend to want production data toremain local and replicate quickly for high availability (favoringavailability over disaster recoverability), sourcing a productioncluster is much more effective in terms of the short term goal whilestill not affecting the long term goal.

Referring to FIGS. 6A and 6B, are a schematic flow chart diagramillustrating an embodiment of a cluster family selection and cooperativereplication method of the present invention. The method 600substantially includes the steps to carry out the functions presentedabove with respect to the operation of the described apparatus andsystem of FIGS. 1-4. In an embodiment, the method is implemented with acomputer program product comprising a computer readable medium having acomputer readable program. The computer readable program may beintegrated into a computing system, such as the cluster manager 320and/or hosts 210, wherein the program in combination with the computingsystem is capable of performing the method 600.

The method 600 starts and in step 605, copying process begins. Forexample, outside data objects in City A needs to be replicate in City B(e.g., FIG. 2B).

In step 610, a control determines whether a cluster receiving the copyrequest is a cluster family member. If not, in step 615, the volume iscopied without performing cooperative replication. For example, thecooperative replication module 415 may manage the copy request withoutdelay or priority changes.

In addition, the cooperative replication module 415 may select at leastone family member in the family to pull the data across a distant linkor network. The selection may be performed once it is determine that thecluster is a family member and no other family members have pulled thedata across the network.

If this is a cluster family member, in step 620, a control determines ifone of the other family members has already completed copying thisvolume. If yes, in step 625, copying the volume is given a lowerpriority and placed back into the queue since one of the other familymembers has already copied the volume.

If one of the family members has not already completed copying to thisvolume, in step 630, a control determines if another family member isactively copying this volume. If yes, in step 635, the priority forcopying this volume is lowered and there is a delay before going backinto the queue. The delay before sending the copy request back into thequeue is to ensure, for example, that another family member activelycopying the volume has not encounter any problems copying the volume.

In step 630, if there is no other family member actively copying thevolume, then in step 640, a control determines if another family memberis also ready to copy this volume, but is not actively copying at thistime. If no, in step 645, a control determines whether this other familymember not actively copying at this time should inherit the copyrequired flags. If yes, in step 645, this cluster lowers copy priorityand delays going back into queue.

If no in step 645, method 600 moves to step 655 and this family memberwins the tiebreaker between the two cluster members and inherits thecopy flags. Accordingly, in step 645, a control determines which familymember will be designated to inherit the copy flags. The non-designatedfamily member lowers copy priority and delays going back into queue(e.g., step 650).

Returning to step 640, if another family member is not ready to copythis volume, then in step 655, a control determines that in this clusterfamily there is only one family member ready to copy the volume anddesignates that family member as the cluster to inherit the copy flagsand complete the replication.

It should be noted that in step 640, a control may determine there isanother family ready to copy and not actively copying at this time, butas indicated in step 645, determine the other cluster will not inheritthe copy flags. Accordingly, the cluster in step 640 would inherit thecopy flags as illustrated in step 655.

In step 660, the designated cluster that inherited the copy flags instep 655, completes copying.

In step 670, a control clears the copy required flags at the sourcecluster and cooperates to cumulatively bring the family to consistencyby setting copy required flags for family members of the cluster family.

In step 675, other family members of the cluster family complete theircopying and their copy required flags set in step 655 within clusterdesignated to inherit the copy flags are reset.

FIGS. 1-3 may be illustrative of a multi-cluster configuration. In amulti-cluster configuration or (grid configuration), from a microcodeperspective, each cluster may be unaware of its relationship and rolesto itself and other clusters and thus, work equally independent of allother clusters. For example, when two or more clusters are configuredglobally-remote from one or more production clusters, they may replicateindependently by ‘pulling’ data across the remote network. Because theclusters have no relationship awareness, they cannot operate in the mostefficient way based on their role and/or distance from other clusters.

Additionally, in a multi-cluster configuration, the means of selecting acluster to source a volume during mount processing and the ability forclusters to honor volume replication is greatly impacted by thisunawareness to relationship. For example, the production cluster maychoose a globally-remote source cluster over a metro-remote cluster formount and/or copy processing. The globally-remote cluster is much lessefficient due to the distance of the network between the clusters.

The implementations of the present invention may resolve these issues bybring relationship awareness among family members and families in amulti-cluster or gird configuration. Additionally, implementing thepresent invention may improve performance, efficiency, and optimizationof the date copying and/or replication. For example, cooperativelyreplicating into a family in order to achieve cumulative familyconsistency N times faster as well as utilizing only 1/Nth of thecumulative network throughput may improves efficiencies and performanceby reducing the overall time to become both DR consistent and HAconsistent versus having each cluster independently sourcing from thesame remote production cluster.

Referring to FIGS. 1-6, the implementations of the present invention mayinvolve software, firmware, micro-code, hardware and/or any combinationthereof. The implementations may take the form of code or logicimplemented in a medium, such as memory, storage and/or circuitry ofhierarchical storage node 315, where the medium may comprise hardwarelogic (e.g. an integrated circuit chip, Programmable Gate Array [PGA],Application Specific Integrated Circuit [ASIC], or other circuit, logicor device), or a computer readable storage medium, such as a magneticstorage medium (e.g. an electronic, magnetic, optical, electromagnetic,infrared, or semiconductor system, semiconductor or solid state memory,magnetic tape, a removable computer diskette, and random access memory[RAM], a read-only memory [ROM], a rigid magnetic disk and an opticaldisk, compact disk-read only memory [CD-ROM], compact disk-read/write[CD-R/W] and DVD).

Those of skill in the art will understand that changes may be made withrespect to the methods discussed above, including changes to theordering of the steps. Further, those of skill in the art willunderstand that differing specific component arrangements may beemployed than those illustrated herein.

While the preferred embodiments of the present invention have beenillustrated in detail, it should be apparent that modifications andadaptations to those embodiments may occur to one skilled in the artwithout departing from the scope of the present invention as set forthin the following claims.

What is claimed is:
 1. A method for cooperative replication of multipleclusters, the method comprising: arranging at least one subset of themultiple clusters into family members of a cluster family; negotiatingbetween cluster family members to determine which cluster family memberis in the best position to obtain at least one outside data object fromat least one cluster outside of the cluster family; selecting one familymember of the cluster family to obtain the outside data object; andsharing the outside data objects among the cluster family so that eachcluster within the cluster family is consistent with respect to outsidedata objects, wherein each family member in the cluster familyreplicates 1/Nth volumes and N represents a total number of clusterfamily members in the cluster family.
 2. The method of claim 1, furthercomprising creating a relationship among cluster families based on atleast one of a cluster relationship and role.
 3. The method of claim 1,wherein the family member with a consistent source maintains a volume incache in order to make the volume readily available to other familymembers for peer replication.
 4. The method of claim 3, wherein thefamily member maintaining the volume in cache for the other familymembers relieves the outside cluster from maintaining a copy in theoutside cluster cache.
 5. The method of claim 1, further comprisingcooperatively serializing all replication into the cluster family. 6.The method of claim 1, wherein a first family member to replicate 1/Nthvolumes informs the outside cluster that the first family member willmaintain the volumes for the cluster family members and relieves theoutside cluster from maintaining the volumes in the outside clustercache.
 7. The method of claim 1, further comprising a plurality ofcluster families wherein at least one family member from each clusterfamily replicates a volume.
 8. The method of claim 1, further comprisinga domain comprising a plurality of clusters from the multiple clustersthat cooperate to make sure at least one family member from each clusterfamily replicates a volume and the remaining clusters surrenderreplication requirements.
 9. A system for cooperative replication ofmultiple clusters, the system comprising: a network; a plurality ofsites in communication over the network, each site comprising at leastone host and a storage system comprising a plurality of clusters, eachcluster comprising at least one tape drive configured to access volumesstored on magnetic tape, at least one tape volume cache, and a clustermanager configured to execute computer readable programs using aprocessor and a memory, wherein the software readable programscomprising: a creation module configured to setup and arrange a group ofclusters into family members of a cluster family; and a cooperativereplication module configured to select a family member to cooperativelyreplicate data from at least one cluster outside of the cluster familyinto the cluster family and achieve cumulative family consistency,wherein each family member in the cluster family replicates 1/Nthvolumes and N represents a total number of cluster family members in thecluster family.
 10. The system of claim 9, further comprising creating arelationship among cluster families based on at least one of a clusterrelationship and role.
 11. The system of claim 9, wherein the familymember with a consistent source maintains a volume in cache in order tomake the volume readily available to other family members for peerreplication.
 12. The system of claim 11, wherein the family membermaintaining the volume in cache for the other family members relievesthe outside cluster from maintaining a copy in the outside clustercache.
 13. The system of claim 9, further comprising cooperativelyserializing all replication into the cluster family.
 14. The system ofclaim 9, wherein a first family member to replicate 1/Nth volumesinforms the outside cluster that the first family member will maintainthe volumes for the cluster family members and relieves the outsidecluster from maintaining the volumes in the outside cluster cache. 15.The system of claim 9, further comprising a plurality of clusterfamilies wherein at least one family member from each family replicatesa volume.
 16. The system of claim 9, further comprising a domaincomprising the plurality of clusters wherein the clusters cooperate tomake sure at least one family member from each cluster family replicatesa volume and the remaining clusters surrender replication requirements.17. A computer program product comprising a non-transitory computerreadable medium including a computer readable program, wherein thecomputer readable program when executed on a computer causes thecomputer to: group a plurality of clusters into family members of acluster family; determine which family member is in the best position toobtain outside data objects from a source, the source included in atleast one cluster outside of the cluster family; select one or morefamily members of the cluster family to obtain the outside data objects;cooperatively replicate the outside data objects into the clusterfamily; and share the outside data objects within the cluster family sothat all copies of the outside data objects within the cluster familyare consistent, wherein each family member in the cluster familyreplicates 1/Nth volumes and N represents a total number of clusterfamily members in the cluster family.
 18. The computer program productof claim 17, further comprising creating a relationship among clusterfamilies based on at least one of a cluster relationship and role. 19.The computer program product of claim 17, wherein the family member witha consistent source maintains a volume in cache in order to make thevolume readily available to other family members for peer replication.20. The computer program product of claim 19, wherein the family membermaintaining the volume in cache for the other family members relievesthe outside cluster from maintaining a copy in the outside clustercache.